Skip to content

[CI] Cross-platform — Part 2: ARM/Spark workflow#5698

Merged
hujc7 merged 70 commits into
isaac-sim:developfrom
hujc7:jichuanh/windows-spark-ci-cartpole
Jul 10, 2026
Merged

[CI] Cross-platform — Part 2: ARM/Spark workflow#5698
hujc7 merged 70 commits into
isaac-sim:developfrom
hujc7:jichuanh/windows-spark-ci-cartpole

Conversation

@hujc7

@hujc7 hujc7 commented May 20, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Adds an arm-ci job to the existing Docker + Tests workflow for NVIDIA DGX Spark self-hosted runners. It builds and tests on the same ARM runner because ECR is not wired for arm64.
  • Adds marker-driven ARM test selection through CI_MARKER, plus a shared kitless Cartpole state/camera smoke used by ARM and installation CI.
  • Extracts Docker dependency-hash and registry-login setup into shared composite actions used by the local Docker and ECR build paths.
  • Makes Docker dependency-cache keys platform-specific and bounds stale ARM cache tags on the long-lived Spark runner.

1. ARM/Spark CI

1.1 The arm-ci job shares the existing changes and config gating. It builds Dockerfile.base for linux/arm64, runs the arm_ci-selected tests, and then runs the shared Cartpole smoke.

1.2 OVRTX and OVPhysX are installed explicitly for the rendering probes. Missing ARM runtime dependencies fail instead of being hidden by architecture-specific skips.

1.3 The job uses the same workflow and shared actions as the amd64 jobs. ARM enables 14-day stale dependency-cache eviction because its runner has persistent local Docker storage and no ECR handoff.

1.4 The arm_ci marker run passes test-k-expr: not ovphysx, the same per-file -k expression the x86 rendering-correctness-kitless job applies on public pip-index runs. The ovphysx-backed rendering params require OVPhysX >= 0.5.1, which has no aarch64 wheel.

2. Shared CI utilities

2.1 _lib/compute-deps-hash computes one dependency-cache identity from install inputs, the resolved base-image digest, and target platform. Both docker-build and ecr-build-push-pull consume it.

2.2 _lib/setup-docker-config centralizes the credential-helper workaround and optional NGC login. It is idempotent when the ECR action delegates to docker-build.

2.3 run-tests forwards both upstream's per-file test-k-expr and this PR's ci-marker inputs into the test container.

3. Shared Cartpole smoke

3.1 source/isaaclab/test/install_ci/misc/cartpole_training_smoke.py is the common prepared-environment probe for state Cartpole training, camera observation/reward validity, and camera Cartpole training.

3.2 CLI and wheel installation tests execute the same probe inside their newly installed environments; ARM executes it inside its prepared image.

3.3 The base image installs git-lfs so RSL-RL can snapshot repository state from a mounted LFS checkout.

4. Cross-platform support

4.1 AppLauncher derives EXP_PATH from the installed isaacsim package when the variable is missing, while leaving existing configured environments unchanged.

4.2 tools/conftest.py supports CI_MARKER as a file-selection and pytest marker filter parallel to ISAACSIM_CI_SHORT, exact node-ID selection, and per-file -k filtering.

Known problems

  • The three rendering modules apply arm_ci at module scope, so ARM collects all 32 cases per module. The ovphysx-backed cases are now deselected via test-k-expr: not ovphysx (section 1.4), leaving 15 Newton cases running per module. Without the filter, the extra ovphysx-ovrtx cases in test_rendering_cartpole_kitless.py pushed a single process past the OVRTX per-process SyncScopeId limit (~15 renderer creations), crashing HydraEngine initialization on the 16th env; this also reproduces on x86, so the failure is not ARM-specific. Narrowing ARM to the two intended representative tests is still pending.
  • Current develop pins OVPhysX 0.5.2+head.f62c22207c, while this job installs the publicly available ARM 0.4.13 wheel. With the not ovphysx filter the ARM rendering selection is Newton-only, matching the x86 public pip-index fallback; revisit if an aarch64 OVPhysX >= 0.5.1 wheel or wheelhouse becomes available.

5. Series

Part 2 of [CI] Cross-platform —. Depends on #5695 (Part 1). The Windows workflow is #5700 (Part 3).

Test plan

  • Merge current develop (5f2db9902baa724427b7aa654835aa559fce64e8).
  • Run ./isaaclab.sh -f before commit and before push.
  • Collect the refreshed ARM rendering selection locally: 96 items (32 per rendering module), confirming the current over-selection.
  • Deselect ovphysx rendering params in the ARM job (test-k-expr: not ovphysx), mirroring the x86 kitless public pip-index fallback.
  • Narrow ARM rendering coverage to the two intended representative tests.
  • Run the two selected tests and shared Cartpole smoke on Spark/aarch64.
  • Confirm the refreshed Docker + Tests workflow is green.
  • Trigger daily-compatibility before merge.

@github-actions github-actions Bot added isaac-lab Related to Isaac Lab team infrastructure labels May 20, 2026

@isaaclab-review-bot isaaclab-review-bot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


Update (df786d2): Reviewed incremental diff since 5cbba72 (1 file changed, 1 commit).

New commits reviewed

Commit PR Description
feba370 #5826 Defer pxr loading in isaaclab.sim, .assets, .scene, .sim/utils
207f319 #5844 Log Kit version info on AppLauncher startup
c54e512 Scope test tags to arm_ci only; misc review fixes
ea8ecd6 #5891 Split tasks into core/contrib
acb09f2 Merge remote-tracking branch origin/develop
5cbba72 Move cartpole smoke test into test/core/ after tasks split
df786d2 Raise cartpole perception smoke-test timeout for cold shader cache

Latest change (df786d2)

Raises test_train_cartpole_perception timeout from default (presumably matching state test) to 1800 s to accommodate first-run shader compilation on cold GPU cache. Adds a comment explaining the reasoning. Minimal, sensible change — shader compilation on first camera-enabled run can easily exceed 600 s on some hardware.

Full analysis (unchanged from prior update)

1. Deferred pxr imports (#5826) ✅ — All 9 modules now defer from pxr import ... into function bodies. Type hints stay under TYPE_CHECKING. Fixes Kit startup crash when env-cfg parse pulls pxr before AppLauncher.

2. Kit version logging (#5844) ✅ — Simple _log_kit_version_info() added to AppLauncher. Prints Kit version, kernel version, and git hash to sys.__stderr__.

3. Tasks core/contrib restructuring (#5891) ✅ — Massive but mechanical: direct.* / manager_based.* → flat core.* / contrib.*. Gym IDs unchanged. Docs updated. Changelog fragment present.

4. CI/test adjustments ✅ — Test markers narrowed, cartpole smoke test moved, perception timeout raised for cold cache.

Previous concern status

Issue Status
P1 (conftest.py OSError handling) ✅ Resolved
P2 (TAB delimiter) ✅ Resolved

No new issues found. LGTM from bot. 🤖


Update (e6184ab): Branch synced with develop (merge commit, 114 upstream commits incorporated — no PR-specific code changes). Previous LGTM still stands. ✅

Foundation for cross-platform CI. Registers four pytest markers
(windows, windows_ci, arm, arm_ci), teaches AppLauncher to recognize
them in argv so they do not leak into Isaac Sim's argparse, and moves
the AssetConverterBase USD scratch directory from a hardcoded
/tmp/IsaacLab to tempfile.gettempdir() for cross-platform compatibility.

Tags source/isaaclab/test/deps/test_torch.py and test_scipy.py with
the new markers so they are selectable by future cross-platform jobs.

Workflow files (arm-ci.yaml, windows-ci.yaml) ship in follow-up PRs.
@hujc7
hujc7 force-pushed the jichuanh/windows-spark-ci-cartpole branch 2 times, most recently from 00966f4 to ca8fa07 Compare May 20, 2026 23:30
@hujc7 hujc7 changed the title [CI] Experimental IsaacLab install probe on Windows via uv [CI] Add ARM/Spark CI workflow May 20, 2026
@hujc7 hujc7 changed the title [CI] Add ARM/Spark CI workflow [CI] Cross-platform CI 2/3: ARM/Spark workflow May 20, 2026
@hujc7 hujc7 changed the title [CI] Cross-platform CI 2/3: ARM/Spark workflow [CI] Cross-platform: ARM/Spark workflow May 20, 2026
@hujc7 hujc7 changed the title [CI] Cross-platform: ARM/Spark workflow [CI] Cross-platform — Part 2: ARM/Spark workflow May 20, 2026
@hujc7
hujc7 force-pushed the jichuanh/windows-spark-ci-cartpole branch 4 times, most recently from 4b8ecb8 to 3e904c3 Compare May 21, 2026 04:43
Mirrors build.yaml's spirit but stays minimal for the aarch64 path:
  Tier 1 (gates none — continue-on-error): general-arm, install-arm, kit-launch-arm
  Tier 2 (meaningful, marker-filtered): kitless-arm, determinism-arm

Every job sets continue-on-error: true while the aarch64 runner setup
stabilizes. Every pytest invocation passes --timeout=N --timeout-method=signal
so a single hung test cannot consume the whole job slot. Inline scripts
use set -e to fail on the first nonzero return.

Tags three test_rendering_*_kitless.py files plus test_differential_ik.py
and test_operational_space.py with the arm_ci marker so the Tier 2 jobs
can select them via pytest -m arm_ci.
@hujc7
hujc7 force-pushed the jichuanh/windows-spark-ci-cartpole branch from 3e904c3 to 98a5b68 Compare May 21, 2026 05:32
@hujc7
hujc7 force-pushed the jichuanh/windows-spark-ci-cartpole branch 7 times, most recently from b7862d0 to 48819bf Compare May 21, 2026 07:33
- source/isaaclab/setup.py: gate pytetwild==0.2.3 with
  `platform_machine != 'aarch64'`. No aarch64 wheel on PyPI; source build
  fails because the transitive `geogram` dep hardcodes `-m64` in its
  CMakeLists. The single call site at sim/schemas/schemas.py already
  lazy-imports it with a clear "install manually" message, so aarch64
  users keep everything except automatic volume-deformable tetrahedralization.

- docker/Dockerfile.arm-ci: new lightweight Dockerfile that layers
  cmake / build-essential / git + EULA env vars + a bash entrypoint onto
  the multi-arch Isaac Sim base image. Replaces the previous inline
  `docker run + apt-get install + docker commit` chain in arm-ci.yaml.

- docker/Dockerfile.base: reverted the libgmp / libmpfr / libeigen3 /
  libcgal / libboost additions from the arm64-conditional apt block —
  those were only needed by pytetwild's fTetWild build, which we no
  longer install on aarch64.

- .github/workflows/arm-ci.yaml: build via docker/Dockerfile.arm-ci
  instead of the inline apt-install-and-commit pattern. Test steps no
  longer need to re-specify --entrypoint or EULA env vars on every
  docker run.
@hujc7
hujc7 force-pushed the jichuanh/windows-spark-ci-cartpole branch from 48819bf to b3d7f7c Compare May 21, 2026 07:39
hujc7 added 2 commits May 21, 2026 07:44
Forces run_docker_tests=false in build.yaml's changes job so all
heavy test jobs skip via their existing if-gate. Saves CI runner
time + cost during ARM CI iteration on PR isaac-sim#5698. Must be reverted
before final review.
hujc7 added 2 commits June 1, 2026 22:05
The develop merge brought in isaac-sim#5891 (split tasks tests into core/contrib),
which relocated every test file under test/core/ or test/contrib/ and left
only shared helpers + conftest at the test-suite root. The new
test_cartpole_training_smoke.py predated that split and was the lone
test_*.py still at top-level. Cartpole is a core task, so move it under
test/core/ to match the new layout, and bump its _REPO_ROOT path depth
from parents[3] to parents[4] (matching the sibling core/ tests).
test_train_cartpole_perception timed out at the default 600s on a
cold-cache Spark runner: the first camera-enabled run compiles shaders
(~600s) before training even starts, which alone consumes the budget.
It passed earlier only because that runner had a warm shader cache.
Give the camera case a 1800s timeout (the state case keeps 600s).
@hujc7

hujc7 commented Jun 1, 2026

Copy link
Copy Markdown
Collaborator Author

Why are we creating yet another separate CI for ARM instead of just letting some of the tests from run on ARM? I vote against that.

Should be fixed.

@hujc7
hujc7 marked this pull request as ready for review June 2, 2026 18:04

@AntoineRichard AntoineRichard left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure about the git action things

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not run this similarly to the other training tests and just add a windows/arm test list like we have the short / long list?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The goal is to delegate config to per test/file level, while keep workflow more stable if I understand the question correctly here

@hujc7 hujc7 added this to Isaac Lab Jun 5, 2026
@hujc7 hujc7 moved this to In review in Isaac Lab Jun 5, 2026
hujc7 added 3 commits June 11, 2026 05:33
…park-ci-cartpole

# Conflicts:
#	pyproject.toml
#	tools/conftest.py
The develop merge removed the versioned Isaac-Cartpole-Direct-v0 and
Isaac-Cartpole-Camera-Direct-v0 gym registrations in favor of the
unversioned Isaac-Cartpole-Direct and Isaac-Cartpole-Camera-Direct ids.
The smoke test still requested the -v0 names, so every run failed at gym
registration with DeprecatedEnv before training started, breaking both
the arm-ci and isaaclab_tasks jobs.

Point the test at the current ids and correct the docstring: the camera
variant now also registers an rsl_rl_cfg_entry_point, so the old
rationale for choosing rl_games no longer held.
hujc7 added a commit that referenced this pull request Jun 20, 2026
…base) (#5695)

## Summary

Base PR for cross-platform CI on Windows and NVIDIA DGX Spark
(ARM/aarch64 Linux). No user-facing API change.

- Registers `windows_ci` and `arm_ci` pytest markers in pyproject.toml.
- AppLauncher argv strip recognizes the new markers so `pytest -m
windows_ci` (etc.) does not leak into Isaac Sim's argparse.
- AssetConverterBase USD scratch dir: hardcoded `/tmp/IsaacLab` →
`tempfile.gettempdir()` for cross-platform compatibility.

This base PR ships only the marker registration and the shared
scaffolding. Tagging specific suites with these markers, and the CI
workflows that consume them, ship in the follow-up parts below — that
keeps the marker/scaffolding change reviewable on its own and lets the
consumer PRs rebase cleanly once it lands.

## Prior art and credits

This work is a derivative of and builds directly on two prior PRs by
other contributors:

- #3900 by Kelly Guo (kellyguo11), against `main`, 2025-10-31 — "test
windows CI". Designed and introduced the cross-platform marker scheme
(`windows`, `windows_ci`, `arm`, `arm_ci`), tagged ~50 existing tests
across the codebase, added the AppLauncher argv-strip pattern so the new
pytest `-m` filters do not leak into Isaac Sim's argument parser, and
pioneered the Windows CI job wiring. The marker design and AppLauncher
behavior in this PR are taken directly from #3900.
- #3845 by Alexander Poddubny (nv-apoddubny), against `main`, 2025-10-27
— "Adds basic CI verification on Windows". Built the original Windows
self-hosted runner infrastructure: `isaaclab.bat` entry script, the
initial `.github/workflows/build.yml` Windows job, and the
self-hosted-runner labeling conventions this PR's downstream Part 2/Part
3 workflows still rely on.

Both PRs target `main` and have not been merged. This PR rebases that
work onto `develop` (where the active development happens — `main` is
behind by hundreds of commits), narrows the base PR to just the marker
scheme plus shared scaffolding (so the marker/runner plumbing can be
validated end-to-end before tests are tagged and re-expanded), and adds
an `arm_ci` marker variant + ARM/Spark runner support that #3900/#3845
did not include.

Credit for the architecture goes to the authors above; this PR is the
rebase + minimization step.

## Series

PRs prefixed with `[CI] Cross-platform —` all contribute to bringing
Isaac Lab CI up on Windows and NVIDIA DGX Spark runners. This PR is the
base everything else builds on. Current siblings:

- Part 1 (this PR, base) — Markers and shared scaffolding
- Part 2 — #5698 — ARM/Spark workflow (`.github/workflows/arm-ci.yaml`)
- Part 3 — #5700 — Windows workflow
(`.github/workflows/windows-ci.yaml`)

Siblings depend on this PR's marker scheme; the same scaffolding lines
are carried in their branches and dedupe on rebase after this lands.
Additional follow-up parts can be added later without renumbering
existing ones.

## Test plan

- [x] `pytest --markers` lists `windows_ci` and `arm_ci` (registered, no
`PytestUnknownMarkWarning`).
- [x] AppLauncher argv strip drops `-m windows_ci` / `-m arm_ci` before
Isaac Sim's argparse runs.
- [x] `./isaaclab.sh -f` (pre-commit) passes.
- [x] Merged latest `develop`: the markers list is the union with the
newly added `device_split` marker, and `--capture=no` argv stripping
from develop is preserved.
try:
import isaacsim as _isaacsim_for_paths
except ImportError as e:
raise RuntimeError(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this path only triggered when we are launching with isaac sim? just want to make sure we don't throw this error if we try to run a kitless path

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe that's determined in simulation context related utilities. In kitless path, applauncher should be skipped.

@@ -0,0 +1,80 @@
# Copyright (c) 2022-2026, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this one similar to some of the cartpole tests we have under source/isaaclab/test/install_ci/cli/test_install_rl_uv_env.py? if we could consolidate some of the tests, that'd help keep files more organized.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's a limitation that install tests needs to be in the same folder. the current approach is to keep the test in the install_ci folder, but keep it in separate file such that other CI can use it.


print(f"ovrtx version: {ovrtx.__version__}")
except ImportError as exc:
if platform.machine() == "aarch64":

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ovrtx and ovphysx seem to provide aarch64 wheels - https://pypi.nvidia.com/ovrtx/ https://pypi.nvidia.com/ovphysx/

hujc7 added 5 commits July 1, 2026 23:44
Share the kitless Cartpole state, rendering, reward, and camera training probes across installation and ARM CI. Install the published OV runtimes on ARM and remove architecture skips so missing dependencies fail visibly.
Use the config job's image tag for ARM builds and isolate local dependency caches by target platform. Clean up merge artifacts and align the artifact action with current develop.
Use RSL-RL for both shared training probes so every validated install\nscope has the required framework. Install Git LFS in the base image so\nRSL-RL can record repository state from mounted checkouts.
…park-ci-cartpole

# Conflicts:
#	.github/actions/run-tests/action.yml
@hujc7
hujc7 requested a review from pascal-roth as a code owner July 3, 2026 03:57
hujc7 added 2 commits July 6, 2026 21:30
Refresh the ARM CI branch with current upstream changes and preserve both the per-file pytest filter and CI marker inputs in the shared Docker test action.
The arm-ci job ran the ovphysx-backed rendering test params that every
x86 kitless configuration deselects via TEST_K_EXPR. These params
require ovphysx >= 0.5.1, which has no aarch64 wheel, and running the
13 extra ovphysx-ovrtx combos in test_rendering_cartpole_kitless.py
pushed the process past the ovrtx per-process SyncScopeId limit (~15
renderer creations), crashing HydraEngine initialization on the 16th
env and failing the job with a missing test report.

Mirror the public pip-index fallback of rendering-correctness-kitless
by passing the same k-expression to the arm_ci marker run.
@hujc7
hujc7 requested a review from kellyguo11 July 8, 2026 16:49
echo "🔵 Local deps tag: ${LOCAL_DEPS_TAG}"
echo "LOCAL_DEPS_TAG=${LOCAL_DEPS_TAG}" >> "$GITHUB_ENV"

if docker image inspect "${LOCAL_DEPS_TAG}" >/dev/null 2>&1; then

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this guarantee that it will always use the latest isaac lab code? the agent picked up a concerning behavior that I'm not too sure if it's true or not

Commit A: build dependencies + source A → cached image
Commit B: source changes, dependencies do not

Current behavior:
cached image containing source A → renamed as Commit B image

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

image is deduped w.r.t dependency hash. source code is always mounted.

needs: [changes, config]
if: needs.changes.outputs.run_docker_tests == 'true'
timeout-minutes: 60
continue-on-error: true

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would this prevent us from getting an accurate signal on whether the tests passed or failed?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It just means that if the job failed, merge is not blocked. If we are confident, this can be made required test.

@hujc7
hujc7 merged commit 7dd722b into isaac-sim:develop Jul 10, 2026
38 checks passed
@github-project-automation github-project-automation Bot moved this from In review to Done in Isaac Lab Jul 10, 2026
hujc7 added a commit to hujc7/IsaacLab that referenced this pull request Jul 10, 2026
Reconcile tools/conftest.py with isaac-sim#5698 (ARM/Spark): adopt the generalized
ctx.ci_marker / run_individual_tests(ci_marker) rename while keeping the
multi-GPU inject_shard_select plugin hook.
kellyguo11 pushed a commit that referenced this pull request Jul 22, 2026
…rmittent hangs (#6581)

# Description

## 1. Summary

Removes the `arm_ci` marker from
`test_rendering_shadow_hand_kitless.py`. Since the test landed on arm-ci
(2026-07-10, #5698) it has been implicated in 15 of 81 concluded arm-ci
runs (~19%) and 15 of that job's 18 failures — including 3 failures on
`develop` itself — typically stalling mid-file after several passing
render cases until the per-file timeout kills the job. x86 coverage is
unchanged: the file still runs in `rendering-correctness-kitless` with
its full backend matrix.

## 2. Failure data (all concluded runs since 2026-07-10)

| Job | Concluded | Failures | Implicating this test |
|---|---|---|---|
| `arm-ci` (blocking on PRs) | 81 | 18 | 15 (3 on `develop`) |
| `rendering-correctness-kitless` x86 (continue-on-error on PRs) | 85 |
32 | 10 (3 timeouts, 7 file failures) |

The instability is not arm-exclusive — x86 hits the same file at roughly
half the rate — but the x86 job is advisory on PRs while `arm-ci` is
blocking, making this test the dominant source of spurious red arm-ci
checks. `test_rendering_dexsuite_kuka_homo_kitless.py` shows the same
signature on arm-ci at a much lower rate (2 of 18 failures) and stays
enabled for now.

## 3. Test plan

- [x] Pre-commit clean (`./isaaclab.sh -f`)
- [x] arm-ci green on this PR with the file no longer collected

## 4. Out of scope

Root-causing the renderer stall (all observed hangs sit in the ovrtx
segment of the parametrized cases, between per-case env create/teardown
cycles); the two remaining `arm_ci`-marked rendering files (cartpole,
kuka-homo) stay enabled.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

infrastructure isaac-lab Related to Isaac Lab team

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

4 participants